Automating Command Line Applications VisualBasic.Net

Posted: April 29th, 2009 | Author: | Filed under: Software | Tags: , , , , , , , , , , , , , , , , , , | 2 Comments »

A lot of the programming that I do for myself involves the creation of little tools that patch two existing parts of my workspace together so that they work better. For example, I recently had need of an SFTP client that links into a database front end to retrieve a list of files to download. Since the SFTP specification is ridiculously long and convoluted, I decided to try and use WinSCP to take care of the FTP portion of the program instead of re-inventing the wheel. Because I was trying to patch the database layer and FTP client together into a single application, I decided to take a shot at controlling WinSCP directly from the command line so that the two portions of the project were contained in a single application.

Readers who have spent time fooling around in Visual Basic 6 might remember the Shell function that allows a program to launch another process. Generally speaking, if we don’t care about the output of the program, or are just launching something like a browser window, this command is more than sufficient. However in this case, I wanted to capture the output of WinSCP and its exit code so that I could tell if errors had occurred and take appropriate action on them, so I needed something with a little bit more kick.

Since I wrote my code with WinSCP in mind, the rest of the article is going to focus on creating an application that will serve as an FTP front-end that allows the user to upload or download a file to or from a remote SFTP server. That said, the ideas are general and easy to adapt to any other application that supports command-line arguments.

You’ll want to begin by dragging and dropping the executable that you wish to automate directly into your project, and setting its “Copy to Output” property to “Copy to Newer” to ensure that your application can always find it. For WinSCP, that included both the winscp.com and winscp.exe files from the install directory. That done, let’s jump in.

Using System.Diagnostics.Process:

This handy System class provides a beefier version of the Shell functionality, and is really very easy to use, once you get the hang of it. At the top of your class, add the line imports System.Diagnostics so that we can find the pieces that we’ll need with ease.

When automating a command-line application, we first create a ProcessStartInfo object that contains all of the information about the process that we would like to create. Start by declaring one of these:

Dim startInfo As New ProcessStartInfo("winscp.com")

This line creates a ProcessStartInfo object with the path to the application that we’d like to automate as it’s only argument. Next, we set a few properties of the object:

startInfo.Arguments = commandLineArguments
startInfo.UseShellExecute = False
startInfo.RedirectStandardOutput = True
startInfo.CreateNoWindow = True

The first line passes the string commandLineArguments to startInfo, telling it to give them to winscp.com as command line arguments. Command line argument options for most programs can be found on their websites. If you’re interested in WinSCP in particular, it has some great online documentation that you can take a look at. The next three lines tell the startInfo object that it should direct the standard output from the process to a System.IO.StreamReader object where we can intercept it, and that it should run that process in the background, without showing the window to the user.

Next, we create a new Process object, and pass it the information that we just stored in startInfo:

Dim p as New Process
p = Process.Start(startInfo)

This creates a new process for winscp.com, and passes it the command line arguments that we stored earlier in startInfo. Now, we need to catch the standard output from our newly created process and store it somewhere so that our users can see what the application did.

Dim s as IO.StreamReader = p.StandardOutput
While p.Responding
Try
lstOutput.Items.Add(s.ReadLine)
Catch
Exit While
End Try
End While

On the first line, we create a new IO.StreamReader object that will catch the standard output from our hosted process. The while loop tries repeatedly to read a line of output from the application and dump it into a list box while the process is still responding to the operating system. When the application finishes its work, s.ReadLine will return null, killing the loop and allowing the program to continue.

Finally, when a program exits, it returns an exit code that lets the operating system know whether or not it completed its work successfully. We can access this code with the line

p.ExitCode

In most programs, if this returns zero, the program finished successfully. If an error was encountered, this function should return something other than zero, and depending on the program, the other value may have some meaning that you can use as a status check.

Next Steps:

Because this code is purposely general, it should be easy to adapt it to control most any application that provides some kind of command line interface. This method can make for a nice alternative to batch scripts if you need to write batch scripts with some kind of logic in them and don’t feel like installing a python interpreter to get the job done. At the cost of making the code application-specific, you could easily change it to analyze the output from the program and take some action based on those return values.

Downloads:

As always, the source is available for download. Check out Tyler Burton’s Hash Verifier application if you want to ensure that you’re getting the same copy of the code that I claim you are.
Source Code: Click Here
MD5 Hash: 4B5D543E193CAC2B4B9477727E17EFB2
SHA1 Hash: ABD8692CBB20D316DBF35FE650399D034DA57697


DRM is a Bitch

Posted: April 18th, 2009 | Author: | Filed under: Software | Tags: , , , , , , , , , , , , , | No Comments »

With nothing important to do this morning (except for all that studying that I should be tackling), I decided to play a favourite game of mine – 1701 A.D. It’s an excellent RTS-style game from Germany in which you play the role of an explorer in the new world. It is your task to build a successful settlement that supports itself and glorifies the Queen. Think SimCity meets Age of Empires, all rendered in pretty graphics and with engrossing game play that will easily make your entire day fly right by. Unless, of course, you try to install and run the game on Windows Vista:

1701drm

Alright, so, the installer must have failed. I went and ran DrvSetup_x64.exe from the game disc, and got another fantastic error message:

tagesdrm

Well, that’s handy. A Google and a half later, I found a slew of angry posts about Tages copy protection on various message boards. I gather that Tages essentially installs a copy-protection driver to your system that isn’t signed by Microsoft. This is an issue under Vista, and the system refuses to install the driver for security reasons. You can reboot and tell Vista to not require that drivers be signed by tapping F8 at boot and selecting “Disable Driver Signature Enforcement” from the boot menu, but according to these posts, you have to perform this action every time you wish to play the game, which is a big security hole as well as a big pain in the ass.

I found a link to an updated version of the drivers at the Tages website that were signed by Microsoft, and installed with no issues, finally allowing me to run a game that I legitimately paid for instead of simply pirating. While I appreciate the situation that studios and distributors find themselves in regarding piracy, I can’t help but be indignant when confronted with copy protection schemes that punish the honest customer. Between my mother and I, our family has purchased three copies of this game, and every previous installment of the franchise. She wouldn’t have been able to figure out the issue with the drivers. Why weren’t the drivers distributed on the game disc signed by Microsoft? Why did the game not automatically update the drivers after validating a legitimate install? Why was there no post about this issue on the game website?

I have to admit that while looking for a solution, I did consider torrenting a cracked version of the game instead of bothering to fix my legitimate copy, which is funny, because the Tages website states that:

The proof of TAGES™’s effectiveness is undeniable as illustrated on various web sites. All major competitors have been hacked and the hackers have made generic cracks available for free. Anyone can break into these systems and produce illegal copies. With TAGES™ there will never be a generic crack, and there will never be one-to-one copies. It is physically impossible.

An interesting claim, since a quick Google search shows quite a few illegal versions of the game available for torrent. At the end of the day, while I cannot condone piracy, this kind of nonsense really gets me angry. I understand that piracy is pushing developers away from the PC platform and toward the more secure Console systems. I understand that piracy can destroy the user experience of a game; but I demand that if a company is going to take steps to prevent piracy, they do it well, and don’t inhibit the actions of legitimate users of paid copies of their product. You wouldnt knowingly push a game to market that had a bug that made it unplayable – so why do DRM schemes get a free pass?


Understanding the Impact of Technology on Your Personal Privacy

Posted: April 7th, 2009 | Author: | Filed under: Education, Software | Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , | 3 Comments »

In short, the concept of personal privacy in communications is possibly the most important right guaranteed to those living in a free society. It is also the single most undervalued freedom in all of Western Society. The right to say what you want to whomever you want puts governing bodies to task, enables rebellion, and ensures that those wishing to sway public opinion have to work hard to demonstrate the value of the opinions that they are trying to impress upon society.

It is thus unfortunate that technology often seems to hinder our ability to ensure personal privacy – at the very least, it makes it easy to ignore the man behind the curtain. Unless one is actively aware of the risks and works to prevent them, most common methods of technological-based communication, including Facebook, cell phones, text messages, instant messaging, web surfing and email all represent massive leaks in personal privacy. These should not be taken lightly, no matter the size of your tinfoil hat. And so, in no particular order, here are some things that you should be aware of when communicating in everyday life.

  1. Facebook and Web 2.0 Privacy:
    While this ubiquitous website has often been accused of selling your information, I find it more strange that it’s users are surprised by the idea that a website owned by a corporation would attempt to monetize the only resources immediately available to it: the information of it’s users. To me, the more scary aspect of Facebook is the slow leak of information that it inevietably causes. Like a small memory leak in an application, it isn’t a huge problem in the short term; but given time, you lose more and more control of your information as it is perused, tagged, linked to, and otherwise aggregated by the website and it’s users.

    As an example, consider the following situation: You go out drinking with friends, and do something stupid. Compromising pictures are taken, uploaded, and tagged by somebody at the bar. What do you do? Well you can un-tag them, but somebody could simply replace the tag, or mention your name in the comments. Even so, anybody who recognizes you could immediately figure out who is in the photo. You can demand the image be taken down, but are reliant on the original poster complying. And even then, who is to say how many people saved local copies of the image, saw it in their news feed, viewed the gallery, or were otherwise linked before it was removed?

    Simply put, Facebook is an easy way to lose control of your personal information. Consider that anybody with a developers license (which is free) has full access to this entire API from any application that they create. The incentive to create malicious trojan applications that steal and sell off information is there. The tools with which to do it are there. And the gullible users who gladly contribute thousands of dollars worth of personal information are there. So even if Facebook is ill-deserving of the allegations of selling users’ information (doubtful), any application that you add can easily present the same danger.

    When using a Web 2.0 site like Facebook, Twitter, or any other site that asks the user to post personal information on a profile, it is advisable to take a quick scan through the Terms of Service (ToS), End User Licensing Agreement (EULA), and Privacy Statement (PS) of the site. Some sites, like Facebook and even Tetris Online, post outrageous claims to user data in their ToS. While I am unaware of any court case that has established a ToS, EULA, or PS as a binding legal contract between website and user, at the time of this writing, it is safe to assume that sites could attempt to act on these ‘agreements’ should user’s violate them enough times, unknowingly or otherwise.

    So what can you do? Wean yourself off the koolaid. Do you really need 300 friends with whom you will never interact in real life? How many of those applications that you’ve installed and websites that you’ve joined do you really use? Is it entirely necessary to list your favourite movies, music, books, last 10 jobs, and your educational information where anybody with an internet connection could conceivably access them? I made the decision to get rid of my Facebook account a few months ago, and have never looked back. After a week, the only thing that I missed was Tetris Friends, which I later found out is available elsewhere anyway (although I haven’t registered a user account  – see Jake’s comment on the Tetris Online ToS below for the reason why). If nothing else, consider taking a stroll through the myriad of options available from the settings page of your favourite social networking site and limiting the access of non-friends to your account.

  2. Cell Phones and GPS:
    Remember the nineties movie slogan? “Shit, he’s on a cell phone. Those are untraceable!” Yeah right. By default, every cellular phone connects with two or more cell towers at any given time, and chooses the one with the strongest signal to transmit and recieve data from. This allows the phone to easily transition between towers without dropping calls while on the move. As a consequence, as long as your cellphone is on, in addition to the knowledge of what towers your phone is within range of,  the phone company can locate the handset to within roughly 1-kilometer of it’s actual location by triangulation. Further, new phones often include a GPS chip that allows you to use mapping applications and geo-tag your photos. If active, the GPS chip can also transmit the location of your phone, often without your knowledge.These tracking features have many positive uses, such as enabling authorities to immediately locate the source of 9-1-1 calls, and allowing business to track the location of their employees in an effort to optimize scheduling. Unfortunately, they also have their downsides – smart phones often come with mapping applications like Google Maps that allow the user to get directions to any destination that update with respect to their location in real time. Who is to say that the almighty Goog isn’t recording all of that data, and linking it with the web-browsing habits and any email received on the same handset? While the process would certainly be undertaken in the name of increased ad-targeting abilities, having all of that relational data lying around can have dangerous side-effects if it is misused, misplaced, or simply sold.The problem of cellphone location is one inherent to the system – the ability to locate phones on the network is a natural side-effect of the way the technology works. There is little that a user can do to prevent their phone from being triangulated. GPS however, is another matter. Many phones give users the options to turn off GPS functionality, or even to limit access to the GPS radio to certain applications. Since I am paranoid, my blackberry is set to disallow Google Maps access to the GPS radio except when I explicitly allow it. This prevents the application from unintentionally spewing my location to Google without my knowledge. As previously mentioned, cameras in newer cellphones that also have GPS can record location information into images taken by the device. Users can turn this functionality off by default, which is a good idea if you intend to upload the photos to social-networking sites or other easily-accessible locations.
  3. SMS Text Messages and Instant Messaging:
    At the end of 2007, an astounding 74% of cell phone subscribers used the SMS text messaging features of their phones. In the book ‘How to be Invisible’ by J.J. Luna, the author reveals that federal law in the USA requires that ‘all billable information [regarding a text message] be maintained for ten to fifteen years,’ including the message contents, date and time of sending and receipt, and the phone numbers of both sender and receiver. Remember the warnings about putting revealing information on the back of a postcard? The same applies to text messages, except that post cards aren’t kept on file by the postal service.Unfortunately, there is no easy solution to the SMS problem. Like post cards, users are best to simply limit what they say via these channels, as they are unencrypted, heavily logged, and contain plenty of identifying information.

    The privacy situation surrounding Instant Messaging programs like MSN Messenger, Yahoo Messenger, and even IRC is in a similar state of disrepair. In order to ensure that MSN Messenger simply works on any machine, regardless of your network situation, all messages are sent from your computer, through a single connection to Microsoft servers, and then forwarded to the intended recipient(s). Messages can be logged on your machine, that of the recipient, or even at the Microsoft servers. Further, all messages are sent in unencrypted plain text that any server along the path from your computer to the recipient can log and store. By the distributed nature of the internet, the very thing that makes it so powerful, the path between any two machines generally consists of 10-15 intermediate hops. (You can check the virtual ‘distance’ between yourself and various institutions in many countries at this website.) That means that between you and your friend, there are 20-30 computers and the Microsoft servers, all of which are capable of logging anything that you say in your conversation.

    Luckily, this problem is far more easily solved than that of SMS messaging. Third-party messenging clients like Pidgin allow you to connect to multiple networks (like MSN, Yahoo, Gmail, and even Facebook chat) at once, and offer an optional plugin called Off the Record (OTR) that can automatically encrypt any messages sent between you and a client that also has OTR running. It is easy to install and mindless to use, and should be a standard feature in every commercial instant messaging application. The only downside to Pidgin is that it looks ugly on Windows machines, but this is offset by it’s plugin abilities, and the fact that it can replace multiple IM clients.

  4. Web Surfing and Internet Connectivity:
    Many people don’t realize what the act of viewing a web page actually is. When you load up this page, your computer contacts my web server, requests the page, and begins to download it to your machine, and then processes and displays the page in your web browser. That means that when you look at this page, all of it’s text, images, and other content are stored in a folder on your computer called the browser cache.

    Along with the history of visited pages that many browsers keep, this information can be used by any person with access to your machine to figure out what web pages you have recently viewed. Further, many web pages leave a file behind on your computer called a ‘cookie’ that contains information that allows web sites to ‘remember’ who you are, which lets them store things like your user name and password, your preferences, or the things in your shopping cart. Again, these files can show people with access to your machine not only what sites you have recently visited, but with what account you logged into them, and potentially, what you did while logged on to the site. You can easily clear the cache, cookies and history from most browsers, or choose not to save them at all.

    Additionally, because the internet is just a massive network of computers, any time you request a page, that request and all of the content that you download from the server hosting the page can travel through multiple servers, and can potentially be logged at any one of them. Further, many internet service providers keep detailed logs of your web browsing activity that authorities or unscrupulous employees can gain access to and misuse. Lastly, in the age of widespread digital piracy, many providers employ a technology called deep packet inspection to determine what your computer is uploading and downloading while connected to the internet. This technology looks inside the messages that your machine sends, determines their contents, and whether or not they should be blocked or limited. By it’s very nature, it also has the ability to snoop on any unencrypted data that you are sending, including your web requests and instant messaging conversations.

    Protecting your information online is a tough thing to do. Of primary concern is the browser program that you use to view web pages. Older browsers like Microsoft’s Internet Explorer 6 have major security holes that can be used by nasty websites to steal your personal information or to install annoying programs on your computer without you doing anything out of the ordinary. Make sure that you have the latest version of your browser of choice installed. Secondly, be careful about what kind of information you give to websites. Do you really need accounts on websites that you use once a month? Putting your name or email address up on these sites can increase spam email, and lead to identity or data theft issues – all of the issues raised during the discussion about Facebook and Instant Messaging apply doubly here. For example, if searching for a job, are resume sites like Monster.ca really necessary? Putting your resume (which contains a bunch of personal information and all of your contact information) up online can lead to some devastating consequences. Finally, make sure that you have updated virus protection and firewall software installed and running on your computer at all times, and turn the machine off when you aren’t using it. If you’re really concerned about your online privacy, look into Tor, a program that encrypts your web traffic and forwards it through a bunch of random servers all over the world so that intermediate servers have no idea where the request is coming from or what data was transferred.

  5. Email:
    All email communications should be considered in the same category as Post Cards and SMS Text Messages. They are unencrypted, used worldwide, and sent through hundreds of servers that all have the ability to snoop or store copies along their journey from machine to machine. Further, webmail addresses like those available from Gmail or Windows Live store your email on their servers (sometimes indefinetly), where the messages and the information that they contain are out of your control and suceptible to snooping by authorities or unscrupulous employees.

    To protect yourself while using email, you should limit the amount of sensitive business or personal information that is sent via unencrypted channels. Further, look into PGP, a (usually) free protocol that can encrypt or digitally sign all of your communications so that others cannot tamper with them. Plugins are available for most commercial email programs, although the best one that I’ve seen is the enigmail plugin for Mozilla’s Thunderbird application. Microsoft Outlook does not ship with default PGP functionality, and most of the third-party plugins that I’ve used are a pain at best and non-functional at worst. Lastly, try to limit your use of webmail, or at the very least, the amount of information that you leave on the remote servers. I use Gmail, but have Microsoft Outlook set up on my desktop which downloads all of my email, saves it locally, and deletes the copies from the server after 30 days.

  6. Bonus Section: Safely Deleting  and Protecting your Files:
    While not strictly a communication issue, many computer users don’t understand how the process of deleting a file on their computer actually works. When you delete a file on Windows, it is removed from it’s original location and sent to a folder called the Recycle Bin so that you can restore it in case you deleted it accidentally. However, even when you empty the Recycle Bin, the file is not physically removed from your machine. In order to save time, Windows simply marks the file as deleted, but never actually removes the data from your hard drive. If, at a later time, the system needs that space, it will over-write the file. But if your computer has a large hard drive that you never fill, chances are that the file can live on in the ‘empty’ space of your hard drive for years to come. Once marked deleted, many freely and commercially available programs can restore most or all of the file’s contents so long as they haven’t been overwritten by new files.

    Because of this functionality, you should always assume that when you delete a file on your computer, it is for all intents and purposes, still available to anybody who cares to look for it. However, you can ensure that the file is safely deleted by using a program called a File Shredder that overwrites the file with random data, making it nearly impossible to ever recover. I would reccommend a free application called Eraser that allows you to shred any file directly from the right-click menu in windows, and can be scheduled to shred the contents of any folder or all of the free space on your hard drive at regular intervals.

    A file shredder can be used to improve your security by securely deleting the contents of your recycle bin, free space on your hard drive, internet browser cache and cookie files, old email, and the porn that you downloaded that you don’t want your wife or boss to find on your machine. It’s easy to set up, integrates directly into Windows, and works without a second thought. Just beware – once a file has been shredded, it’s gone for good. Make sure that you aren’t going to need it before shredding it.

    Finally, to prevent people unwarranted access to your sensitive data, look into a full-disk encryption application like TrueCrypt. It encrypts your entire hard drive and refuses anybody access until they enter a secret password that you set. Windows Passwords are good for preventing access to your machine, but if an attacker removes your hard drive and pops it into another computer, the Windows password doesn’t help you in the slightest. Full-disk encryption however, makes it extremely hard, if not impossible, to get at your files unless you personally unlock the machine.

    The more expensive versions of Windows do offer a file encryption system called Encrypting File System (EFS) that can optionally encrypt your files and folders with a combination of symmetric- and public-key cryptography, similar to the system used by PGP. One potential problem with the scheme lies in the fact your Windows password and decryption password are one and the same. When you log in to Windows, the operating system transparently decrypts any files that are requested by applications. As long as you have a strong Windows password that you change every so often, this can be a good solution; however, by default, TrueCrypt encourages the use of two separate passwords and demands the first before Windows even boots, which can be far more secure (as long as the attacker chooses to boot Windows and not a separate OS from a CD or DVD drive if they get past the TrueCrypt password).  Secondly, EFS does not provide full-disk encryption. Instead, it allows the user to choose which files and folders they would like to encrypt. This is generally alright, except that many programs leave digital litter around your hard drive that may not be encrypted under this scheme. For example, if you encrypt a Word document and then open it, Word can create a series of unencrypted temp files on your drive while you work on the file. Unless you wipe the free space on your drive on a regular basis, this may not be desireable when working on sensitive. If an attacker were to pull the hard drive from your machine, they could gain access to any files that you had not expressly set as encrypted by EFS. For this reason, full-disk encryption provides a better out-of-sight, out-of-mind solution that is guaranteed to protect all of your sensitive data.

Alright, if you’re still here after that massive article, I hope that you found it informative, enlightening, and easy to understand. Technology can seem tough and scary, but it doesn’t have to be that way. With a little bit of well-placed education, anybody can understand and improve the security of their communications in an effort to protect themselves from identity theft, unwanted intrusions, and overzealous authorities.

As this kind of stuff is a hobby of mine, I will be happy to answer any questions raised by or not covered by the post – leave me a comment!

Cheers,

Jonathan

Edit: Thanks to Tyler for pointing out that Enigmail for Thunderbird is an optional plugin, and is not included by default, as well as the information about Window’s EFS technology. Also thanks to Jake for pointing out the importance of reading the ToS of your favourite websites.