Windows 98 can be a bit particular about VirtualBox compatibility. The same is true for Windows 95. First of all, make sure to read the official documentation on the topic here: https://www.virtualbox.org/wiki/Guest_OSes
Also try upgrading to 5.1.x and making sure you're not trying to boot in UEFI mode or something like that which Windows versions that old don't support.
One more tip: Don't try to allocate more than 256 MB of RAM to Windows 98. It can't address more than 512 MB anyway, and it is known to behave oddly at its limit, producing random "Out of Memory" errors for no obvious reason.