Colour & Shape: Using Computer Vision to Explore the Science Museum Group Collection
Introduction
Online museum collections provide a treasure trove of objects to explore. They also allow visitors to view a much larger proportion of a museum’s collection than they could see by visiting the museum in person; the majority of many museum collections are kept in storage.
This article analyses a selection of the Science Museum Group Collection. We examined over 7,000 photographs of objects from 21 categories. The categories were selected on the basis that they contained large numbers of everyday or familiar objects. These categories range from photographic technology to time measurement, lighting to printing and writing, and domestic appliances to navigation.¹
The photographs allow us to study the form of objects — their shape, colour and texture. The insights gleaned can be used to enrich a museum’s catalogue and expand the ways of searching through the online collection. More broadly, this analysis shows how a photographic dataset can shed new light on a museum’s collection.
This research was funded by the Creative Industries Policy and Evidence Centre (PEC).
Colour
The most common
Amongst the 7,083 objects that we analysed, the most common colour was a dark charcoal grey. The collage below shows objects that contain large amounts of this colour. While this grey appears in over 80% of the photographs that we studied, in most cases it only makes up a tiny fraction of the pixels in any one photograph.
Changes in colour over time
The image on the left shows how the colours of objects have changed over time. Each object was assigned to a year group (spanning 20 years), based on the earliest year associated with the object. We then calculated the mix of colours amongst the objects in each group.
The video below provides a more detailed picture of how colour has evolved. The objects are ordered by their earliest date and each frame shows the mix of colours in overlapping groups of objects.
The most notable trend, in both the chart and the video, is the rise in grey over time. This is matched by a decline in brown and yellow. These trends likely reflect changes in materials, such as the move away from wood and towards plastic. A smaller trend is the use of very saturated colours which begins in the 1960s.
While things appear to have become a little greyer over time, we must remember that the photographs examined here are a just a sample of the objects within the collection, and the collection itself is also a non-random selection of objects. Moreover, these trends will continue to change as new objects are acquired.
The colours within a single object
The video below illustrates the huge array of colours that can be found within a single object. The object in question is a Century Model 46 plate camera, dating from 1900. It can be found in The Kodak Collection at the National Science and Media Museum in Bradford.
Colours in old and new technology
The images below contrast the range of colours in a telegraph from 1844 with an iPhone 3G mobile phone from 2008–2010. The telegraph is a Cooke and Wheatstone double needle telegraph which relays messages by using needles that point to different letters on the dial.
The wide range of colours in the telegraph comes in large part from the mahogany wood used in its construction. But the colours also come from its shape (the rounded pillars reflect light and create shadows) as well as its age (the wear and tear creates colour variations). In contrast, the metal and plastic materials in the iPhone give much less variation. It also has a more basic shape and is in better condition.
Hidden colours
The dataset of photographs can also be used to identify tiny amounts of colour across large numbers of objects. As an example, the collage below shows a number of pocket watches (and one pendulum clock) which were all found to contain a small number of blue pixels. These shades of blue are rarely seen in objects from the 1800s. The row of squares beneath each photograph shows the exact shades that can be found in each watch.
In several cases, the blue comes from the hands of the watch. For others there is a blue tinge to the screws at the back of the pocket watch. This is due to a process called ‘bluing screws’ which is carried out to make the screws resistant to rust. The blue is an oxide layer which comes from heating the screws.
The most colourful objects
The collage below shows objects which contain the greatest number of unique colours. For most, it is the packaging around the object that is colourful, rather than the object itself. The two earliest objects both date from 1920s — one is a cigarette packet and the other consists of four H chargers and two films for a Pathescope 9.5mm cine camera. Most of the other objects date from the 1980s and several are computer or board games, which reflects a combination of factors: the advent of computer-based design and printing, the rise of consumerism, and a concerted effort by the Science Museum to collect games around that time.
A closer look at phones
The Science Museum Group Collection contains hundreds of phones, dating from the late 1800s to the present day. The video below uses a sample of these phones to show how the colour of this object has changed over time. The five most common colours in each phone are displayed behind the phone and are then added to the chart on the right-hand side.
Interestingly, some of the earliest telephones share the same black and silver colour scheme that is seen today in many smartphones. In contrast, phones from the 1960s, 70s and 80s covered a broader range of colours. The ‘greying’ began in the late 1980s, with the introduction of the brick phone.
Colour, shape & texture
An alternative museum map
The form of an object is not just a function of its colour but also its shape and texture. This section considers all three aspects and highlights some of the more unique-looking objects within the collection.
To create the map below, machine learning was used to automatically group the objects. Similar-looking objects (based on their photos) were placed near each other. The border around each image indicates the earliest year associated with the object. A larger version of the map is available (63.2MB).
The most common shape
Almost all recent objects (which have pale borders and date from the late 1940s) are clustered together in the north-western area of the map. The objects in this area are all cuboids (or ‘box-shaped’), and range from cigarette packets to televisions, and from mobile phones to computer games. These recent objects adjoin a cluster of much older items (with dark red borders). These older objects include boxes for money, weights and snuff, and they share the same cuboid shape. To the machine learning algorithm, a modern laptop looks similar to an old money box.
While the box appears to be the most common shape for recent objects, there are two interesting exceptions. The first is the table telephone, and a cluster of these can be found in the middle of the map. These phones have a much more complex shape, with curly cords and handsets.
The other exception is modern translucent objects, which form a cluster in the south-eastern corner of the map. Objects in this area boast a variety of shapes (owing to their decorative purpose and the malleability of their materials) but their common feature is translucency, with many glass plates and bottles.
Unique groups of objects
There are several ‘islands’ in the map. These are groups of objects that are visually distinctive from the rest of the collection. One island consists almost entirely of typewriters. The typewriter’s unique appearance stems in large part from its components being visible to the user. Moreover, these components have unique shapes, from the ribbon which is wrapped around spools, to the platten which is cylindrical. Most typewriters also contain several levers, such as the carriage return, which protrude either side of the machine. All these features give typewriters their distinctive appearance.
‘Were it possible by an edict to forbid for one week the use of these wonderful machines the whole business world would be cast into such inextricable confusion as could hardly be conceived’ 6th January 1901 (New-York tribune)
There are a number of other ‘islands’ in the map which each contain unique-looking groups of objects within the collection. These include a collection of viscose skeins, as well as a set of ancient Egyptian and Syrian weights.
Unique individual objects
In addition to finding unique groups of objects, the map can also be used to identify individual objects that have a highly distinctive appearance. A selection of these objects are shown below. They were found by measuring the distance of each object to its five nearest neighbours.
The collage includes several domestic appliances: a marmalade cutter, a heater designed to recycle waste hot air from a fireplace, and a rotary cheese grater. There are also two art deco objects: a photo frame and a loudspeaker. And there are two Kinora viewers, which are similar to ‘flip books’. The user looks through a viewer while turning a handle which causes the picture to flip over.
Perhaps the two oddest objects are the sample of turf and what appears to be a blue rock. These two objects are connected because they both relate to recycling. The artificial turf is an example of a product made from recycling old Nike shoes. And the blue rock is actually broken and scrap glass (called cullet) which is added to the furnace when making new glass. It is a way of recycling the glass and it speeds up the process, thereby saving on fuel.
Final thoughts
The Science Museum Group’s open dataset of photographs allows us to learn more about their collection. Our preliminary analysis suggests that everyday objects may have become a little greyer and a little squarer over time. While only time will tell, it does highlight a challenge for museums who must engage visitors with these ‘black boxes’. Collectively examining objects also allows us to identify and celebrate the most distinctive objects, from typewriters to table telephones. And it lets us spot small patterns that we might have otherwise missed, such as the bright blues within 19th century pocket watches. As computer vision methods continue to improve we will be able to extract further insights from online collections and learn more about the objects that fill our lives.
Methodology
The aim of this piece was to show the value of online collections and to encourage the exploration of other online collections. Before launching such an exploration, there are several challenges to bear in mind, which we describe below.
First, it may not be possible to analyse an entire collection. Not every object within the Science Museum Group Collection has been photographed, and others have only been captured in black-and-white. We also had to exclude any photograph whose background colour was not uniform. In these photographs it is too difficult to accurately distinguish the colour of the object from its background. Non-uniform backgrounds tended to arise where the object was large (such as a machine) and the photograph had been taken in situ. The uniformity of the background colour was judged by measuring the variation in colour around the very edge of the photograph.
A second challenge is accurately identifying the colours of an object within a photograph, and excluding background colours. We did this by extracting every pixel from a photograph and then dropping any pixel whose colour was too close to the most common edge colour. This approach presumes that the object itself never touches the edge of the photograph. The key variable is the threshold distance below which a pixel is considered to be ‘too close’ to the background and dropped. If the threshold is set too high we may be underestimating the amount of near off-white colours within objects (most backgrounds are white). If the threshold is set too low we may accidentally include some of the background colour in our analysis. For some parts of the analysis we also had to round the colours, to reduce the number of unique colours and allows us to display them in the charts.
The third challenge relates to measuring the similarity of objects, or more accurately, the similarity of their photographs. We used a convolutional neural network model (VGG16) to extract feature vectors for each image. The dimensionality of these vectors was reduced using Principal Components Analysis (PCA), followed by t-Distributed Stochastic Neighbour Embedding (t-SNE). These give a simplified approximation of each object and they don’t account for the scale of an object; objects of different sizes (but related outlines) may be judged as being similar to each other. In order to clearly display the t-SNE mapping the images were placed into a grid. The placement of objects within the grid aimed to minimise the distance of the object from its true location, but there will be some distortion (particularly in dense areas where there are many similar objects).
Finally, there are two broader points to consider. The colour of an object will be influenced by its surroundings, how it is displayed and how it is photographed. Two photographs of the same object, from different angles or against different backgrounds, may generate different colour profiles. And it is important to remember that the results shown above will continue to change as the collection evolves, and as more objects within the collection are photographed.
Footnotes
¹ We excluded specialist categories, such as ‘Surgery’ and ‘Railway posters, notices and handbills’. We also excluded very broad categories such as ‘Art’ and ‘Documents’ as well as categories that mainly contained parts of objects, such as ‘Electrical Components’.
About the author: Dr Cath Sleeman is a data scientist and Head of Data Visualisation in the Creative Economy and Data Analytics team at the charity Nesta. In her work she explores novel sources of big data and creates unique data visualisations to tell data-driven stories.
