Introduction

Malware like OceanLotus have used RSA-256 to hide strings. Darkhotel used RSA to hide code. For fun, some crackmes used RSA-32 or RSA-64 for simple keygen challenges. The RSA cryptosystem uses two exponents (or keys) and a modulus derived from two primes. One of the exponents (the public key) is for encryption and the other (private key) is for decryption hence the reason it’s called “asymmetric encryption”.

However, there exists a (bad) way to use RSA for symmetric encryption that isn’t advisable (it’s not really encryption..i know). It’s made possible by the underlying function called modular exponentiation. While it’s not a good idea to use for encryption, we can take advantage of it for simple masking that’s less complicated to implement than RSA itself.

RSA Cryptosystem

Since the masking is based on RSA, let’s recap how it works.

The selection of e and calculation of d in key generation ensures that d is not an involutive exponent. Without these steps, there would be no security provided.

Finding Involutive Exponents

If you decide to modify the existing RSA algorithm, you just need to ensure the exponent is involutive for it to work as a masking function.

Composite

'''
    RSA-16 using involutive exponent
    
    e=1249
    n=32881
    message=77
    masked=3426
    unmasked=77

'''

import sympy

bits = 8

# Step 1: Choose two distinct prime numbers p and q
p = sympy.randprime(2**(bits-1), 2**bits)
q = sympy.randprime(2**(bits-1), 2**bits)

while p == q:
    q = sympy.randprime(2**(bits-1), 2**bits)
    
# Step 2: Compute n
n = p * q

# Step 3: Compute Euler's totient function φ(n)
phi_n = (p - 1) * (q - 1)

# Step 4: Choose integer e such that e^2 ≡ 1 (mod φ(n))
e = None
for e in range(2, phi_n):
    if ((e * e) % phi_n) == 1:
        break
  
# demo
message = 0x4D

print(f"e={e}")
print(f"n={n}")

print(f"message={message}")

masked = message ** e % n
print(f"masked={masked}")

unmasked = masked ** e % n
print(f"unmasked={unmasked}")

Prime

A simpler approach is just generating a random prime number. For every prime number, there exist one or more exponents which you can find with the loop used in step 4. All we do here is find the first one and exit the loop.

'''
    modexp using involutive exponent
    
    e=20789
    p=51971
    message=77
    masked=3690
    unmasked=77
'''

import sympy

bits = 16

# Step 1: Choose prime p
p = sympy.randprime(2**(bits-1), 2**bits)

# Step 4: Choose integer e such that e^2 ≡ 1 (mod p - 1)
e = None
for e in range(2, p):
    if ((e * e) % (p - 1)) == 1:
        break
        
# demo
message = 0x4D

print(f"e={e}")
print(f"p={p}")

print(f"message={message}")

masked = message ** e % p
print(f"masked={masked}")

unmasked = masked ** e % p
print(f"unmasked={unmasked}")

Other

For an 8-bit involution, use an exponent of 127,129 or 255 with 257 as a modulus.