Building the data block
Before any RSA math happens, your plaintext is packaged into a data block: the SHA-256 hash of an (empty, by default) label, then a string of zero bytes as padding, then a single 0x01 separator byte, then your actual message.
The OAEP data block (DB)
lHash
SHA-256 of the label (empty string by default) — always 32 bytes.
PS
Zero bytes, padding DB out to a fixed total length.
0x01
A single separator byte marking where the padding ends.
M
Your plaintext message.
Masking twice with MGF1
A random seed is generated, then MGF1 (a mask generation function built from repeatedly hashing seed‖counter) stretches it into a mask the same length as DB. That mask is XORed into DB to produce maskedDB. Then MGF1 runs again, this time on maskedDB, to produce a second mask that's XORed with the original seed to produce maskedSeed.
OAEP's two-round masking
- 1
Generate a random seed
32 bytes, fresh every time you click Encrypt — this is what makes RSA-OAEP non-deterministic.
- 2
Mask the data block
maskedDB = DB ⊕ MGF1(seed) — the seed's randomness now covers the whole message.
- 3
Mask the seed
seedMask = MGF1(maskedDB); maskedSeed = seed ⊕ seedMask — the seed now depends on the message too.
- 4
Assemble EM
EM = 0x00 ‖ maskedSeed ‖ maskedDB — this is what actually gets RSA-encrypted.
The same masking, as a block diagram
Each half masks the other: the seed masks DB into maskedDB, and maskedDB then masks the seed into maskedSeed. The three pieces — a leading 0x00 byte, maskedSeed, and maskedDB — concatenate directly into EM.
OAEP encoding: two rounds of masking
The seed masks DB (top); the resulting maskedDB then masks the seed right back (middle, dashed line shows the original seed feeding that second XOR) — each half depends on the other, which is what makes recovering any part of the message require recovering the entire encoded block.
Then, and only then, the RSA step
Everything above happens before any modular exponentiation. The assembled encoded message EM is what gets raised to the public exponent: C = EMᵉ mod n, using the same textbook RSA formula covered in the RSA module. The base64 string labeled "Ciphertext" in this tool is exactly that C, encoded.
Decrypting: unmask, then check the structure
Decryption computes Cᵈ mod n to recover EM, then reverses the masking (recover seed from maskedSeed, then DB from maskedDB) and checks the result's structure: does it start with 0x00, does the recomputed lHash match, is the padding string all zero bytes, and is there a 0x01 separator in the right place? Only if every check passes is M returned.
This is exactly why the "wrong private key" button in this tool fails outright rather than returning garbled text: Cᵈ mod n with the wrong d produces essentially random bytes, and the odds that random bytes happen to satisfy "starts with a valid hash, has a correctly placed separator, all-zero padding" are astronomically small. The browser detects the malformed structure and throws, rather than silently returning nonsense.