Hail relies on OIDC email claims to verify the validity of a user's domain.

Impact

All Hail Batch clusters are affected. An attacker is able to:

1. Create one or more accounts with Hail Batch without corresponding real accounts in the organization.

For example, a user could create a Microsoft or Google account and then change their email to “[email protected]”. This Microsoft or Google account can then be used to create a Hail Batch account in Hail Batch clusters whose organization domain is “example.org”.

In Google, this attack is partially mitigated because Google requires users to verify ownership of their Google account. However, a valid user is able to create multiple distinct Hail Batch accounts by creating multiple distinct Google accounts using email addresses of the form “[email protected]”.

In Microsoft, this attack requires Azure AD Administrator access to an Azure AD Tenant. The Azure AD Administrator is permitted to change the email address of an account to any other email address without verification. An attacker can create an Azure Tenant for free.

1. The attacker does not have access to any private data (because the new service principals or service accounts are not granted any privileges).
2. If trial Hail Batch billing projects are enabled, the attacker does have the ability to run jobs and thus spend money. An attacker can create as many accounts as Microsoft or Google permit.
3. The attacker cannot impersonate another user because, in Azure, we use the sub from the OAuth2 response, and, in Google, Google does an email verification.

Remediation

1. Apply this patch to prevent third-party attackers from creating accounts.
2. Audit your users list https://auth.example.org/users for user accounts whose login ids are not valid login ids with your identity provider. Delete such users.

A forthcoming change will prevent users from creating multiple accounts using Google’s + email redirection.

Workarounds

None.

References

1. https://trufflesecurity.com/blog/google-oauth-is-broken-sort-of/
2. https://www.descope.com/blog/post/noauth
3. https://developers.google.com/identity/openid-connect/openid-connect#an-id-tokens-payload
4. https://learn.microsoft.com/en-us/entra/identity-platform/access-token-claims-reference#payload-claims

[1] Hail Batch must separately stop using emails and start using the OAuth2 sub in Google. This is a known deficiency. In particular, if an email is re-used by the organization for a new user, the new user could access the old user’s Hail Batch account.