TAIPEI (Taiwan News) — A software company explained to Taiwan News from July 7-10 how it trained artificial intelligence (AI) to find a satellite image of a balloon spotted over the Taiwan Strait that matched the profile of the Chinese surveillance balloon that flew over the U.S.
From Jan. 28 to Feb. 4, a high-altitude surveillance balloon flew over the U.S., causing a diplomatic row between Washington and Beijing. On Feb. 9, a New York Times report indicated that a similar Chinese balloon may have flown close to Taiwan on Sept. 27, 2021, but the Ministry of National Defense (MND) said it was spotted 40 km away from the country's coast.
The report included a photo by Central Weather Bureau (CWB) Director-General Cheng Ming-dean (鄭明典), showing the Judicial Yuan at the bottom left, the Presidential Office Building at the bottom right, and the balloon appearing as a white spot nearly invisible in the middle, far off in the distance. Another photo shows a closer image of the balloon with an oblong panel of instruments tethered below it, similar to the one seen in the U.S.
Photo by Cheng Ming-dean taken on Sept. 27, 2021 at 8:29 a.m. (Cheng Ming-dean photo)
In an email interview from July 7-10, a Synthetaic representative told Taiwan News that the CWB photos provided the starting point for their team's research. Using the firm's Rapid Automatic Image Categorization (RAIC) AI software, team members found what they believe to be the same balloon at latitude 25.35688714 and longitude 120.63860078, which is approximately 57 kilometers northwest of Taiwan in the Taiwan Strait on September 27, 2021.
The size and altitude of the balloon were consistent with the surveillance balloon that flew over the U.S., according to Synthetaic. The size of the balloon’s payload was roughly equal to two school buses, and past balloons matching its profile were determined to be equipped with multiple antennae capable of picking up and geolocating communication signals, such as cellphone transmissions.
In terms of the methods used to search for the balloon, the representative said that Planet Labs operates a constellation of around 200 satellites that scan the Earth every day. Synthetaic works with PlanetScope Monitoring, which collects over 300 million square kilometers of imagery every day and has an archive of high-resolution images dating back to 2009.
Suspected Chinese spy balloon on Sept. 27, 2021. (Facebook, Cheng Ming-dean photo)
Synthetaic’s RAIC platform is designed to rapidly search for massive amounts of visual data, including satellite imagery. When BBC’s Panorama team provided Synthetaic with social media reports of a balloon over Taiwan in September 2021, the team fed RAIC a reference image of a surveillance balloon over the U.S., and queried the relevant area of Taiwan during the relevant time period for matches.
The company said it has not yet conducted any searches since February 2023. Therefore, there is always a chance there has been more activity, "we’d just need to ingest and run the satellite data from February on through our AI."
The representative added that because it used daily pictures to find the balloon, they were not able to provide granular details about its journey, but they were able to trace the origin of the Chinese balloon to be near Hainan Island in the South China Sea.
Regarding the blue, red, and green dots seen in the image, the representative said there are three dots because of the way satellites take photos with different RGB lenses, not always at the same time. Therefore, the three colors demonstrate that the balloon was moving.
The following is Synthetaic founder Corey Jaskolski's explanation from March about how AI was employed:
“I started from a simple hand drawing of how I thought the balloon might look in satellite imagery. Knowing that many satellite-borne cameras take multiple photos through separate-colored filters, separated by fractions of a second, I drew something resembling a red, green, and blue snowman. This would represent the balloon getting captured through these various filters as it moved with the high-altitude winds. I fed that image to RAIC and used it to search imagery taken above South Carolina before the balloon was shot down. Within two minutes, RAIC had returned a positive match.”
Jaskolski said that he was initially skeptical that the image was indeed the spy balloon. He was later led to believe it was the balloon based on its "size, shape, nearby social media reports, and wind maps showing where it was expected to be," and the use of parallax to calibrate the balloon's altitude.
Initial sketch of balloon (top left), followed by confirmed images of balloon that flew over U.S. (Synthetaic image)