I've created a custom script and would like to be able to install it using the API, but that requires that it have a script ID and when I run list installed scripts with the API, it doesn't give me script IDs and my custom script appears in the list. When I run list scripts, I see the sid(s), but my script is not in the list. Is there no way to install a custom script using the API?
The process to install a custom script is same as installing a default script like WordPress.
But as you said that your custom script is not listed by the "list_installed_scripts()" function it means that you have not added the custom script to Softaculous. You may refer to the following guide on how to add a custom script :
http://www.softaculous.com/docs/Adding_Custom_Scripts