This is the second post in a series about applying UX theory in the real world.
I’m a Designer armed with a solid framework for approaching User Experience (UX). I’m trained to research tasks, goals, behaviors and motivations; to create personas, storyboards and scenarios; and to build, test and iterate on prototypes.
At ADstruc, we adopted the Agile approach in product development, which is based upon rapid iterations and constantly adapting to change.
As a designer, this can be somewhat stressful. After all, how can you collect all the information you need to make design decisions when sprints roll by so fast and tickets are sometimes as granular as changing the color of a link?
Previously, I wrote about how you can involve people in your organization in the UX process as a first step toward applying your UX design theory to everyday practice. The next step is adapting your process to work better with the way your product is developed. This post illustrates how my UX role fits within the Agile methodology at ADstruc. This process won’t necessarily work for every organization or product, but I hope it will provide some guidance for marrying product with design decisions and using your UX deliverables as ways to “feed the Agile machine”.
There’s No Need to Fuss
As a UX designer, you collect a lot of feedback about the users of your application and its usability, and make changes to your design where needed. As an Agile adopter, you base your constant iterations on the output of a feedback framework and needs that arise throughout development. Feedback is clearly a common ground and is the basis of any Agile/UX relationship.
Let’s start by taking a look at where UX fits within a more traditional Waterfall model. Product development goes through 4 to 7 different stages, including conception, design, implementation, and maintenance. Each stage has to be completed before the following one can be launched. As a UX designer, you can take charge of the design stage, and do some long and expensive design research which will lead you to find the problem and the perfect solution. There isn’t much room for error; you have to get it right or wait months for another expensive rehaul.
Figure 1: UX within a classic Waterfall model
The Agile manifesto professes to care more about individuals and interactions than processes and tools, more about responding to change than following a plan. As a UX designer, you believe in the huge effect user research and testing have on the different iterations your product goes through. With Agile, you have a bigger opportunity to create faster change through your UX findings and get developers to apply your feedback into actionable user stories.
At ADstruc, design, product management, and engineering all work together in the earlier phases of a project’s conception— during what we call “working sessions”— to come up with a solution to the user story we are discussing. We try to finish most design work for a sprint before it effectively starts. This includes sketches, wireframes and flows, and high fidelity mockups where needed. Some small engineering tickets don’t get working sessions, but most bigger ones that require design input will be a collaboration that happens a few days before the start of a sprint. This way, engineers have the information they need to estimate the complexity of a story and can focus on getting the idea executed during the time allocated to it.
Our design approach is very lean. Since solutions are discussed as a team, UX documents can be at the lowest level of fidelity and don’t require extensive documentation. Personally, I am a big fan of hand-drawn wireframes because they are quickly produced and favor collaboration. I attach them to the story at hand before they’re discussed with the team, and often they are modified during or after the working session. Since I am constantly on the lookout for research data about our users, I can combine the knowledge I’ve built with input from other team members to recommend better solutions.
However, a lot of questions still arise around the loss of sufficient UX data to back up design claims. What happens to interviews, contextual inquiries, personas, and storyboards? Designers worry that they will lose grip of the coherent experience they aspire to create and fall into the so awful world of stacked up features and disordered UI elements. I, myself, am guilty of that.
I’ve been at ADstruc long enough to witness the before and after of going Agile, and I can safely say that the best way to adapt to the change in methodology is gradually, by taking everyone’s existing activities into consideration. By keeping in mind that Agile is a philosophy and not a process set in stone, you can use it to your advantage as a means to structure your work and then redesign it and your process to better work together as one entity.
So how do you make use of all of your design methodology in an Agile environment?
Applying UX Elements to Agile Development
The biggest challenge UX designers face when trying to immerse in an Agile process is getting the time they need to collect information about users. Since user stories are often granular, they don’t always require a dedicated design research effort. And since data is not always plentiful in a small startup, I am constantly on the lookout for opportunities to talk to our users, learn about their motivations and conduct user interviews. This happens in two ways:
- A formal approach to interviewing and contextual inquiries (watching the user in their normal activities), which often requires a tough recruitment process. I try to do these quarterly.
- An ongoing effort attending business demos, client meetings, and keeping in sync with account management and customer support, so that I’m always aware of customer frustrations.
Another form of UX feedback is obtained through Hallway testing in the early stages of feature implementation. This can help find big problems with the interface or confirm the effectiveness of our design direction. We do this by building a quick functioning prototype of our ideas and grabbing a few people at our office that are not involved in the design or development of the feature to try it out and test our hypothesis.
We also build lean versions of features that we test with real users. Redesigning big parts of our application is a highly iterative process that involves feedback directly from customers. I plan and conduct more formal user testing later on in the development phase of bigger releases, usually also quarterly. Since it’s difficult to run a lot of small scale tests, these sessions may need to do double duty. You can focus them on the feature you want to test, and at the same time watch for general usability issues.
Finally, it’s important to keep an eye on analytics. We look at them during the ideation of a feature, when confirming a hypothesis is possible through quantitative data. I also keep a whiteboard updated with the most important information on our users’ browsers and operating systems, so it’s quickly available to the team when we’re brainstorming.
Figure 2: UX contributes to the Agile process by affecting product decisions and feeding the backlog
A Two-Sided Relationship
Evidently, it is possible for UX designers to maintain their UX process throughout Agile development. Not only that, Agile development benefits UX designers, in that it allows them to get their ideas implemented faster. But, on the other hand, UX also plays a big role in boosting the feedback loop of Agile development by constantly feeding design data to it. I’ve mentioned that I use my research findings to recommend better solutions during working sessions. But the results I find and share with the team also contribute to the creation of our design and engineering backlog. User stories are created when a problem, or need, is found in the interface or in our user’s workflow and my role as a UX designer is to find some of these problems. Design research also produces personas that help with the creation of goals and scenarios for sprint tickets. And finally, user testing is one great way to enhance the product backlog. I discuss the results with our product manager who translates them into stories. This way, the feedback I collect has a quick impact on the next Agile iteration.
The way I see it, Agile development and User Experience design are built on the same standards of communication, testing and quick iteration. They work together in symbiosis, making organizations more cognizant of external reality, thus leading to a better product.
Like a lot of designers out there, my academic training covered interaction design processes and methods - the immense benefits of contextual inquiries, observation, and the how-to of user research interviewing. I read a lot of books on the subject and hundreds of blog posts and opinions. But applying it all in the real world is totally different. Even though User Experience (UX) design readings are numerous, well-documented case studies are harder to find.
Over the next few weeks, I commit to showing you how we design products at ADstruc, what we found useful, and how we shaped the process to fit our needs.
This first post is about creating a UX culture within your company. Not only will it get everyone on the team involved and excited about solving user problems, it will also help you utilize everyone else’s very valuable input as part of your workflow.
After being an all-around software developer at ADstruc for over a year, I seized an opportunity to move to what I really wanted to do, UX Design. Hooray! It was exactly what I had spent a long time preparing for. But being a solo act (ADstruc is a small company) was pretty overwhelming. I had never applied UX principles on a full-time basis, let alone lead my own department. I had to establish a structure for myself and get other people to buy into and respect it.
I quickly realized that in order for me to be able to make use of everything I knew about designing interfaces and get people to acknowledge the effort and methodology, I had to stop thinking about it as a solo act and include everyone in the process.
Tip #1: Make everyone your ally.
Working with product management and engineering is a must: kudos to our product manager Jeff who brought this culture to the team. He introduced working sessions that engineers and designers have to participate in every time we brainstorm the design and implementation of a feature. One engineer runs point and is accountable for delivery, but everyone on the team (product management, engineering, UX and visual design) is responsible for contributing to the solution. You also have to work closely with other designers in your team. There is no question that UX and visual design have to work hand-in-hand in creating all mockups and prototypes to deliver to engineers. But working together earlier on in the process, when you’re creating the first quick drafts, gets your ideas to flow more easily.
The bond you form with support and account management is one of your most important resources. I will delve into the details of data collection and user research in the posts to come, but, trust me, gathering relevant data - which is so hard to find in a young startup - is one of your toughest jobs. Working with the people who interact with and support your customers will provide you with valuable insights into problem areas with your interface. It will also help you understand the context in which people are having these difficulties. I ask our account manager Neal to forward me emails he receives from angry or confused customers, and I am included on our company’s support email address. Neal also complements most of my research findings, proof reads my customer journeys, and is a big resource when it comes to creating personas.
Lastly, user experience has to account for and align with business needs. Don’t forget to keep in sync with the C-suite; after all, you’re creating products that serve your users and your business.
Tip #2: Turn some of your UX tasks into a team project.
Now that you have allies, introduce them to your world and include them in the UX process. It will not only shed light on the things you are working on, but also foster user empathy - which should be a core team value. Involving your team in identifying frustrating user experiences will increase personal responsibility toward fixing them.
Customer journeys, personas, research. I’m sure you have integrated a lot of these into your process. But think about the fun other people will have getting involved in something every once in a while that they don’t ordinarily get to do as part of their day-to-day jobs. Doing UX at ADstruc has given me the added bonus of introducing a lot of exciting “fun crafts”, as some people here call them. Markers, colorful post-it notes, notecards. They will make for some fun events at your company!
Personas, for example, are best done as part of a team, especially since you want everyone to use them. I introduced the need for personas soon after I took charge of our UX. I organized a breakfast meeting for the whole team and we came up with persona drafts for the sell-side of our platform. I even cooked the food myself! We produced some really cool documents, and had fun drawing what we imagined our personas to look like. Even as a first pass, they were useful in product development.
Involve your team in usability tests and encourage them to sit in on your phone interviews. Engineers will see firsthand the extent of user frustrations and can use the feedback in day-to-day problem solving. They will also understand the limitations you face collecting data, and can give you great pointers on how to analyze it. Visual designers will witness and appreciate users stumbling over an interface they felt was perfect and they’ll eagerly return to the drawing board with you.
Also, record everything you can, from usability tests to phone calls. Some team members may not have time to sit in on calls but recordings will always be available to watch or listen to.
Tip #3: Be a show-off.
You have the benefit of working on a lot of visual documents, so make them a public display! People will stop and ask you about your work and they will also have great ideas and suggestions of things you can pursue. When we were working on mapping customer journeys, I took over a couple whiteboards so that everyone could see them. I think it got everyone intrigued, a few people asked me about the graphs I was creating, and that lead to several interesting and productive conversations.
These are just a few of the many things I’ve learned at ADstruc about working with people. I would love to hear about your experience and what has worked for your company!
After having pruned proposals for an RFP, our agency users often require a PowerPoint presentation export of the chosen inventory to present to their client. Our software enables them to do so, generating within seconds a PowerPoint that usually takes them hours to put together by hand. These PowerPoints contain maps detailing inventory locations in all the relevant markets, inventory photos, pricing and other relevant information. Recently, one of our users requested us to include labels containing unit numbers for the inventory markers in the coverage maps. Since proposed inventory within a market can often be located very close together, we had to come up with a strategy for labeling these markers while reducing the overlap amongst them so as to maximize readability. Here is a screenshot of a map with inventory labelled but not optimized for readability:
We ended up experimenting with two (local) search AI algorithms to optimize label sizing and placement for maximizing readability. AI search algorithms are used to find optimal solutions to problems that can be modeled in terms of discrete states, and a transition function that can produce allowed (neighboring) states when given a state and a legal transition as input. The set of allowed states for a problem formulates its search space. The algorithm has to traverse the search space efficiently and arrive upon a state that optimizes a pre-defined objective function. Brute-force search algorithms like Depth-first Search and Breadth-first Search are too inefficient to be able to solve problems that have large search spaces.
For the map-marker labeling problem, the objective function that we chose to minimize was the total label overlap (measured in square pixels). A state, in the context of this problem, refers to a map with a certain number (say, n) of labeled markers. Further, a neighboring state is produced by moving a label on any one marker to a different position. Since we allow 10 different positions for a map-marker label, our transition function can produce 10n neighboring states for any given input state (also know as the branching factor). Our search space consists of 10^n states. For 50 markers, the search space has more states than the number of stars in the universe (10^24)! Here are screenshots of the 10 different positions that we implemented for a label on a marker:
The labels have a maximum width of 110px. The large ones have a height of 21px (11px font) and the smaller ones are 14px high (8px font).
The first search algorithm that we experimented with, Hill Climbing, is a greedy algorithm which when applied to our problem, can be summarized in the following steps:
- Given an initial state, pick any two labels that overlap.
- Move one of the labels to a position that reduces the total label overlap.
- Set the resulting state as the initial state.
- Repeat steps 1-3 until the overlap has been reduced to zero or a certain number of iterations have been run.
The second algorithm, Simulated Annealing, is a stochastic search algorithm. It can be summarized in the following steps:
- Given an initial state, pick any two labels that overlap.
- Move one of the labels to a random new position.
- Set the resulting state as the initial state with a probability of where is the difference in “system energy” (total label overlap) between the resulting state and the prior state and T is the cooling schedule (explained below).
- Repeat steps 1-3 until the overlap has been reduced to zero or a certain number of iterations have been run.
Our experimentation methodology was as follows – n markers were placed randomly on a map of size 450px by 450px (the size of the maps in our PowerPoint exports) and the value of n was set to 10, 20, 30, 40, 50 and 60. For each value of n, we ran the experiment 30 times to reduce the bias of any individual run. For each random placement, we ran both algorithms (with a maximum of 200 iterations each) and measured the percentage reduction in label overlap. The average overlap reduction was calculated for both algorithms for each setting of n. The results are presented in the following graph:
The Simulated Annealing algorithm outperformed Hill Climbing by an overall margin of 17% (absolute). As expected, we observed that the latter would often get stuck in local minima – unable to find a globally optimal solution. The former, on the other hand, because of its stochastic nature was able to explore a much larger fraction of the search space and return a minimum that if not globally optimal, was almost so. The following graph which shows data from one of our runs with a map with 50 markers illustrates this point well:
In this case, even though the greedy Hill Climbing algorithm is able to reach a minimum faster, it gets stuck there – unable to find a neighboring state with a lower value of the optimization function to move to. The Simulated Annealing algorithm does not always pick a state with a lower value of the optimization function (as can been seen by the intermittent spikes in the graph) but is eventually able to navigate to a more optimal solution.
Affect of Cooling Schedule on the Performance of Simulated Annealing:
The cooling schedule (T) essentially determines how fast the randomness in the Simulated Annealing algorithm decreases. An ideal cooling schedule has a T that is high when the algorithm starts so that it accepts states with higher “energy” with greater probability. As the algorithm proceeds, and it gets closer to discovering the global minimum, the value of T should reduce so that transitions are made only to states that have lower energy. Consider the following analogy – if you try getting a ball to the lowest point in a box containing a solid material that forms peaks and valleys in the box, you could shake the box vigorously at first (high T), getting the ball to bounce around randomly. As the ball starts entering low valleys you reduce the shaking and let gravity take over (low T), getting the ball to roll into the lowest valley on it’s own.
A variety of functions can be used for T – exponential, logarithmic, oscillating etc. The choice of a cooling schedule function is very specific to the problem at hand. Thus, we performed an experiment in order to determine the optimal cooling schedule for our problem. We generated 30 maps, each with 50 map markers placed randomly and ran Simulated Annealing with seven different cooling schedules on each one. We chose a mix of functions with a variety of slopes and periods. The seven functions were:
Where is the maximum number of iterations for a run of the algorithm (which we set to 200) and is the current iteration number (1 through 200). The periodic exponential functions involve 4 periods of the exponential function (we perform the following assignment before calculating the value of the function: ). All seven functions are plotted in the following graph:
The results obtained from this round of experiments are summarized in the following graph:
The logarithmic function outperformed the others, achieving an average overlap reduction of 85.37% over the 30 runs. The Periodic-Exponential-2 function was a close second with an average overlap reduction of 85.23%. It seems like the algorithm performs best with a schedule that has a high value to start and then decays very quickly (i.e. large slope). The logarithmic function fits the bill. The periodic exponential functions also perform decently well, especially Periodic-Exponential-2. It has high slope, and because it is periodic, it causes the algorithm to discover at least 4 (the number of periods in the function), possibly different, minima. The minima that is the least of the ones discovered can be used as the solution.
Finally, here is the result of running Simulated Annealing with the Logarithmic cooling schedule on the map shown above (overlap reduction of 97%):
And here is the result of running Hill Climbing on the same map (overlap reduction of 84%):
The first one clearly has more unobscured labels, and is more readable.
We were successfully able to apply two search (optimization) algorithms from AI literature on the map-marker labeling problem. Even though the results are specific to our setting, we believe that the Simulated Annealing algorithm would perform well in a generic point-labeling setting.
We held our first every Hack Day (Happy Pi Day!) today at ADstruc. Every month, we’re going to be devoting one day to build a small-but-useful piece of software and open source it. That’s right, open source it! Giving back to the community is a core part of our company culture, and we’re super-excited to express this value through our product development process. This time around we built WaitableButton, a jQuery plugin that can be used to add a waiting state while binding a jQuery $.ajax() object to your buttons. Check out some examples, or download it now!
MongoDB has been the target of a considerable amount of hate from the community of late. Here’s the backstory — in 2008 began a “mass exodus” of sorts towards (immature) NoSQL technologies. Bright-eyed architects began adopting databases like MongoDB without developing a complete understanding of their features and limitations. Over time, they realized the err of their ways. The full implications of the loss of the relational and ACID properties of the RDBMSs that they had abandoned only became clear as their applications grew. Their thoughts of 1000 machine mongo clusters were replaced by the realities of rewriting GROUP BY in code for the twelfth time; these early adopters began ranting and raving against poor lil’ Mongo.
That said, this is not another MongoDB-is-a-PITA-post. Sure, we’ve run into our own share of issues with Mongo. But to us, the most interesting ones have arisen from our use of an ODM (Object-Document Mapping) system as an abstraction of Mongo. We decided to use Mongo because it seemed to be a good fit for early versions of our application – we used its geospatial features heavily, and had a handful of primary domain objects that were more or less disjoint. However, as the application evolved and our data started to look more and more relational, we decided to adopt Doctrine’s ODM (heavily inspired by their ORM which in turn is modeled after Hibernate) to act as a bridge between the document paradigm and the traditional relational paradigm. In a lot of cases, this abstraction works well but in others, it breaks down; Doctrine’s ODM is a leaky abstraction. As Joel Spolsky puts it in his Law of Leaky Abstractions:
"All non-trivial abstractions, to some degree, are leaky."
An ODM is an incredibly difficult abstraction to implement. It attempts to create an ORM-like mapping, which is already conceptually and technically tricky due to the impedance mismatch between the OO and relational paradigms, for a datastore that is very different from a traditional RDB. In some cases, these differences can be worked around in code, resulting in additional complexity and inefficiency. In other cases, the differences remain, and are rendered invisible to the ODM user; the biggest danger of this sort of abstraction lies in the fact that it creates an illusion of similarity. Users of the ODM, like us, fall prey to concluding incorrectly that since it looks like, swims like and quacks like an ORM, it probably is one. Under the hood lies a very different beast. Here are some of the ways in which we were bitten by way of such thinking:
1. Workaround for lack of nested positional operator caused unexpected results
Mongo’s positional operator does not support the ability to update arrays nested within arrays (an issue present since March 2010). However, our domain model consists of many one-to-many-to-many style relationships that are implemented as array embeddings by the ODM at the DB level. For instance, we have a ‘proposal’ object which has a one-to-many relationship with a ‘property’ which in turn has a one-to-many relationship with a ‘message’ which in turn has a one-to-many with ‘attachment’; a proposal document contains an array of property documents and so on… However, since the positional operator can only perform updates up to one-level deep (in this case, to properties only), updates to further levels of nested arrays must be performed by a workaround in code. Because of a bug in the ODM to work around this limitation, we found that when trying to add a message with attachments to a property, the attachments would always get added on to the first message in the array. Going forward, we’re working around this workaround by avoiding more than one level of nesting — normalizing our data across multiple documents in such cases. This comes at the cost of performance, and the cost of increasing the probability of data inconsistency issues (due to lack of multi-document atomic updates i.e., transactions). This inconsistency in how Mongo treats top level documents vs. embedded documents is not only an issue with the Doctrine ODM, but also with the popular Mongoid ODM as well.
2. Unexpected query implementation caused race conditions
In a feature that kicked off multiple asynchronous updates to a certain object, we discovered a bug that seemed to be causing the data corresponding to a certain property on the object to disappear. After a few hours of digging deep into the internals of the ODM, we discovered that the bug was caused by an unexpected implementation of what we assumed was an atomic write. While aware that Mongo doesn’t implement transactional support, we assumed that single-document ODM level updates were mapped to single-document updates at the DB level and hence, were atomic (would be a pretty safe assumption in the case of most ORMs). However, we noticed that while updating the value of a property on an object that is an ArrayCollection type (a Doctrine collection type which wraps native PHP arrays), the ODM executed two queries on the corresponding document – one to unset the property (rendering it empty), and a second to set it to the specified value. This, of course, led to a race condition when multiple processes tried to write/read changes to/from this object. Often, processes that tried to update the object after an update to the ArrayCollection property was initiated by an earlier process, read the object in a state in which the property was unset. These processes wrote the object with the property in the unset state, making it seem like data was disappearing.
3. Emulation of joins causes performance issues
Most ORM’s support the retrieval of an object graph from the underlying database with a single query but since MongoDB does not support joins, the retrieval of an object and its associations involves many roundtrips to the database resulting in a process that is obviously less efficient (admittedly, we are yet to quantify the extent of the performance hit). The number of queries that the ODM must perform in order to retrieve an object graph is more or less opaque to a developer who is not conscientiousness of this limitation of the database — another downfall of this abstraction.
Even though abstractions enable developers to write cleaner code faster, they also increase the risk of their making incorrect behavioral assumptions. This is especially true when abstractions that are generally well understood are recreated for disparate underlying systems. However, we agree with Jeff Atwood in that the most useful (non-trivial) abstractions (also the most leaky) are here to stay. It is our job as programmers to understand the deficiencies of such abstractions and not reject them, for in return they promise us much simplicity.
When we were looking around for a good resource on getting started with Selenium and Google Maps, we found that many of the articles out there were about v2 of the Google Maps API and the few StackOverflow posts only covered the absolute first steps of getting up and running. We’ve grouped together some of the tips and tricks we’ve learned along the way of building out our test suite.
With the proliferation of browser technologies such as Canvas, Google kept optimizing Maps to handle more markers and more layers, faster than ever before. The downside to this (from the Selenium testing perspective) is that many of these optimizations break the normal pattern one uses when writing a Selenium test. Luckily, Google has given us a way to disable these optimizations when creating a marker, by setting optimized to false in the google.maps.MarkerOptionsobject.
Rather than inserting the marker into one of its existing Canvas tiles, this causes Google Maps to render it as individual DOM (Document Object Model) element. Be forewarned, you should only do this for development/Selenium testing purposes, as the performance penalty is quite noticeable. At ADstruc we do this by including the environment inside a globally accessible ADstruc object on the page and then only modifying the google.maps.MarkerOptions object when we’re in our testing environment.
Side note – while Google Maps disables right clicking on elements within the map container, it’s still possible to use Web Inspector to inspect elements within the map by using the magnifying glass and then clicking on the element.
Now that you and Selenium can actually find the markers within the DOM, how do you go about picking out a specific one? Unfortunately, Google Maps doesn’t tie a specific ID onto a marker or provide a way of programmatically accessing the underlying DOM element. This makes sense because, except for cases where the optimizations are specifically turned off, there are no DOM elements to directly reference from the marker object. There are a few tricks we can combine into a serviceable workaround. The first is that all markers belong to the class gmnoprint. However, all of the map controls (zoom, layers, copyright bar, etc.) also belong to this class. Most of these you can turn off when initializing the map, but a few (the copyright bar and report a map error) are always there. If you’re trying to get a count of the markers on the map, it’s a good idea to take a baseline reading of the number of elements that belong to the gmnoprintclass before adding the markers in.
More likely, you are going to want to interact with a specific marker – check if it’s on the map, mouseover it, click it, etc. To do this requires a bit more digging into the structure of a marker element and the commandeering of its title attribute. There are two distinct structures, one for markers with events and another for those without. The markers without events are very simple and the title is attached to the containing div element:
The ones with events have a few more elements to support making the marker image clickable. Here, the events and the title are attached onto the area element:
In terms of being able to set arbitrary data onto the DOM element of a marker, the title element is, currently, our only option. At ADstruc, we don’t use the title attribute in our actual application, so it was easy to take it over for identification purposes. If your application does make use of the title attribute, you’re going to have to implement a switch so that you can safely use it during testing purposes. For our application, we have three types of markers – Points Of Interest (one from the Google Places API and another saved into our system) and Units. All three of these have IDs that are unique, so it was easy for us to simply set the title to <type>-<id> during testing. We’re using Facebook’s WebDriver library, but these examples should be applicable no matter which language you’re using.
Note – you could replace this with an XPath selector as well, however, we’ve always found the regular selectors much easier to debug and XPath under Internet Explorer can run into some speed issues depending on the complexity of your selectors.
Now that you’ve got your marker selected, you can pretty much treat it as you would any other Selenium element. Remember that if you want to trigger events, you’re going to need to select the area element (which is also the one where the title element is attached). We haven’t had the need to interact with polylines just yet, but they may prove to be a more difficult challenge, since there doesn’t seem to be an optimized attribute in the google.maps.PolylineOptions.
If you’re looking to purchase outdoor advertising and haven’t had a chance to check out the most efficient way to do so, click here to schedule a demo with one of our account managers.
At ADstruc we’re constantly importing and updating inventory for our 800+ Out-of-Home media (includes billboards, subway ads, telephone kiosks etc. — check out OOH-101) partners. The inventory data that we receive is usually comprised of excel sheets listing the ad-spaces/units, accompanied by photo sheets containing unit photographs. Over time, we’ve developed tools to help us assimilate this data automatically. The part that has proved most challenging to automate is the generation of crops from the photo sheets to produce images that conform to the dimensions that we use on the site. It’s difficult because the position of the units in the photographs is arbitrary. So, for instance, in cases in which the photograph is taken on approach on a highway, the unit might be placed to the side (left or right) of the photo. Hence it’s impossible to make assumptions about unit position. Here are some examples:
Further, our platform supports a Creative Upload feature which allows an advertiser to preview their creative on a unit before choosing to buy it. In order for the creative image to be superimposed on the unit, its four corners need to be pre-identified (if it is a parallelogram — most OOH units are).
Recently, we developed a billboard detection algorithm (using Python and the open-source image processing library, OpenCV), to identify the unit in a photograph automatically, crop around it and return its four corners.
Thankfully, the units are usually outlined, or are flood-filled in the provided photos – this provides us with a great signal to start. However, since the images are highly compressed, artifacts such as ringing make it non-trivial to detect unit boundaries with great accuracy. We began by thresholding the image — allowing a range of colors resembling the outline/flood color to pass through. We hand-picked a narrow initial range and a wide final range. The algorithm successively backs-off to the wider range until a parallelogram is found. Here’s the binary image obtained from thresholding the first image show above:
Given the thresholded binary image, our first attempt at billboard detection involved the use of the Harris Corner Detection algorithm. However, given the edgy nature of the boundary in binary image, the algorithm ended up identifying a large set of “corners” (points with low self-similarity), not including the actual corners of the shape in some cases. This method was too sensitive to the presence of pixels from extraneous objects. Here the output obtained from applying the Harris Corner Detection algorithm to the binary image shown above (corners are marked by green circles):
Our second attempt involved the use of simple geometry to extract a parallelogram from the binary image. We calculated the center of the unit outline by averaging pixel co-ordinates (assuming that only outline pixels made it through the thresholding step). The corners were then determined by computing the point farthest from the center, the point farthest from this point, the point farthest from the line joining the two points, and lastly, the point farthest from the line joining the two points but on opposite side of the line that the third point lay on. However, this method broke down (quite badly!) in cases when pixels from extraneous objects made it to the thresholded image.
Our third and most successful approach involved the use of many techniques. We began by applying the Contour Detection algorithm given by Suzuki et al. to determine the extreme outer contours in the binary image, using simple chain contour approximation. Then, we applied the Douglas-Pecker Polygon Approximation algorithm to the determined contours. From the computed polygons, we retained only approximate parallelograms (quadrilaterals with parallel sides, or almost so) with a certain minimum area. If more than one parallelogram was found, the one with the largest area was retained. This method proved most successful, and we observed a recall of 80% and a precision of 91%. Given the four corners of the “billboard” it was trivial to find its bounding box, and crop around it. The corners were stored for the purpose of the aforementioned Creative Upload feature.
The outputs from our most successful approach, when applied to the first two images in this post, are shown below (detected billboard corners are marked by green circles):
Next, we’re going to try to apply a supervised machine learning algorithm to this problem while seeding it with the labelled data obtained from the unsupervised technique described here. Stay tuned!