[Info-vax] In need of "good enough" randomness sources, anyone wanna help out? (OpenSSL 1.1.1 beta)

[Richard Levitte]

Den tisdag 1 maj 2018 kl. 09:10:22 UTC+2 skrev Richard Levitte:
> Den tisdag 3 april 2018 kl. 18:00:36 UTC+2 skrev Richard Levitte:
> > [disclaimer: this discusses pseudo-random numbers, not true randomness, so any time you see "random" written, think "pseudo-random".  Bill Gunshannon, please pay attention and don't derail]
> > 
> > Hey,
> > 
> > The randomness machinery in OpenSSL has changed radically for upcoming version 1.1.1.  The implementation is now a block cipher DRBG as specified in https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-90ar1.pdf, based on AES-256-CTR.
> > 
> > To get some kind of seeding, this implementation does like before, and relies on data from an entropy aqcuisition function (see below).  It worked, by delivering just enough entropy (64 bytes at assumed 4 bits of entropy per byte => 256 bits, which is exactly enough for AES-256-CTR).
> > 
> > Now, we've run into a bit of trouble because of a very recent change that adds the possibility of a nonce to the pool.  When no nonce is given, but there's still a nonce size specified, and I quote:
> > 
> >     /*
> >      * NIST SP800-90Ar1 section 9.1 says you can combine getting the entropy
> >      * and nonce in 1 call by increasing the entropy with 50% and increasing
> >      * the minimum length to accomadate the length of the nonce.
> >      * We do this in case a nonce is require and get_nonce is NULL.
> >      */
> > 
> > So we suddenly find ourselves in a situation where 256 bits of entropy isn't enough, the requirement has been elevated to 384 bits (96 bytes à 4 bits of entropy per byte).
> > 
> > The VMS entropy acquisition function can be seen here:
> > https://github.com/openssl/openssl/blob/master/crypto/rand/rand_vms.c
> > 
> > It relies almost entirely on the data returned by sys$getjpiw for these item:
> > 
> >     JPI$_BUFIO
> >     JPI$_CPUTIM
> >     JPI$_DIRIO
> >     JPI$_IMAGECOUNT
> >     JPI$_LAST_LOGIN_I
> >     JPI$_LOGINTIM
> >     JPI$_PAGEFLTS
> >     JPI$_PID
> >     JPI$_PPGCNT
> >     JPI$_WSPEAK
> >     JPI$_FINALEXC
> > 
> > So I'm looking for ideas, are there other items you'd recommend?  Other sources?  I've eyed through the JPI$_ macros, but am frankly getting a bit blurry eyed and don't feel I quite have a grasp of how variable the data for each item is (i.e. how good entropy it can give).
> > 
> > Cheers,
> > Richard
> 
> So I did a bit of experiment and collected more data, most of all quite a lot of the RMI$_ items but also a few more JPI$_ items, at a total of 736 bytes per gathering round and a "random" 1 to 10 second sleep between rounds, fed that into a file that I then fed NISTs minimum entropy estimation program (*)...  and the result was pretty harsh, an estimate of 0.082 entropy bits per 8 bits of data (**), so roughly 1 bit of entropy per 100 bits of data.
> 
> The problem is that a DRBG using AES256-CTR (which is the current new OpenSSL implementation) demands a minimum of 256 bits of entropy, i.e. a little more than 3KiB of data gathered at a maximum pace of 1 736 byte chunk per second.  Yay for waiting about 5 seconds before one can even get some randomness out of the DRBG (***)...
> 
> I'm not amused.
> 
> /Richard
> 
> (*) https://github.com/usnistgov/SP800-90B_EntropyAssessment
> 
> (**) Running the non-iid program, this is what I get:
> 
>     : ; ./a.out ../../../levitte/vms-experiments/entropy-gathering/entropy-stats.bin  8 -v
>     Opening file: ../../../levitte/vms-experiments/entropy-gathering/entropy-stats.bin
>     
>     Running non-IID tests...
>     
>     Entropic statistic estimates:
>     Most Common Value Estimate = 0.975224
>     Collision Test Estimate = 0.902997
>     Markov Test Estimate = 0.410808
>     Compression Test Estimate = 0.811274
>     t-Tuple Test Estimate = 0.0818796
>     Longest Reapeated Substring Test Estimate = 0.0818772
>     
>     Predictor estimates:
>     Multi Most Common in Window (MultiMCW) Test: 100% complete
>     	Correct: 507351
>     	P_avg (global): 0.508671
>     	P_run (local): 0.587891
>     Multi Most Common in Window (Multi MCW) Test = 0.76638
>     Lag Test: 100% complete
>     	Correct: 269907
>     	P_avg (global): 0.271051
>     	P_run (local): 0.347168
>     Lag Prediction Test = 1.52629
>     MultiMMC Test: 100% complete
>     	Correct: 11700
>     	P_avg (global): 0.011977
>     	P_run (local): 0.444824
>     Multi Markov Model with Counting (MultiMMC) Prediction Test = 1.16869
>     LZ78Y Test: 99% complete
>     	Correct: 572107
>     	P_avg (global): 0.573391
>     	P_run (local): 0.615723
>     LZ78Y Prediction Test = 0.699647
>     Min Entropy: 0.0818772
> 
> (***) There's a thought that libcrypto could start a separate thread upon initialization that would gather data in the background, say once a second...  it will take about 5 seconds for it to gather enough entropy at a 736 bytes per second rate.

My gathering program can be found here.  It's sloppy code, but does the job:

https://github.com/levitte/vms-experiments/tree/master/entropy-gathering

Cheers,
Richard