> unlink "file.txt";
> Is there a better/correct way to delete a file?
This way is correct. There are other ways, but there is not a better way.
In fact, this way is better than some of the others, because it will work (assuming the filename is correct) irrespective of platform, and also because it does not invoke an unnecessary external program. The unlink builtin has been in Perl at _least_ since perl 5.003, probably longer, so it is safe to use in code that has to work on crochety old setups.
For the other portion, getting the list of files in a directory, there is always opendir/readdir, or you may also be able to use glob (or even the <> glob operator). If you are doing anything nested, you might also want to look at File::Spec or File::Spec::Functions (depending whether you prefer an object-oriented or plain vanilla call-this-function paradigm, respectively).