One approach would be to use the Hough transform. The Hough transform can
find certain geometric features in an image, such as lines and circles.
The algorithm for detecting straight lines can be outlined as follows:
1) First you filter the image with an edge detection algorithm. Many
different algorithms exists for this purpose. There is one algorithm known
as "Canny's edge detector", which is quite powerful.
2) Now, for each pixel, you need to store the set of feasible lines that can
pass through it somehow. Of course there are infinitely many such lines, so
you need to perform some kind of discretization.
We know from basic algebra that a line can be explicitly determined in the
following parametric form:
y = ax + b
By varying the constants a and b we can find any number of lines passing
through the point (x, y). However, for our purpose, it turns out that the
following parametric representation is better:
rho = x cos(theta) + y sin(theta)
since now we do not need to worry about the fact that the constant a
approaches infinity for completely vertical lines. If we discretize theta
into a fixed number of angles, then we can also determine the corresponding
values of rho for the point (x, y). Now, the trick is to accumulate a
two-dimensional matrix at the positions (rho, theta) for each coordinate (x,
y) that corresponds to an edge pixel. This way certain combinations of (rho,
theta) will appear more frequently when they lie on the same line.
3) Finally you should find the largest values in the accumulated matrix.
These corresponds to the (rho, theta) values that are most likely to
represent a line. Thus, you can now use this information for finding the
corners in a rotated rectangle or some other polygonal object (i.e. by
looking at the intersections of the lines).
Here is a small example application I wrote for Graphics32: