No, Luke Skywalker Didn't Use a Forced Choke in Return of the Jedi
While Luke Skywalker appears to be using a Force choke in Return of the Jedi, both New Canon and Legends confirm that's not the case.

Luke Skywalker appeared to use the dark side power of the Force Choke in Return of the Jedi, but both the new Star Wars canon and the Legendary books suggest that's not the case. Luke Skywalker's battle with the dark side is a huge part of his character arc in the film, and while he eventually resists its allure, he nearly succumbs to it while battling his fallen father . Luke's inner conflict is represented in many overt and subtle details in Return of the Jedi, but he doesn't actually use the Dark Side's famous Force choke at any point in the film.
When Luke Skywalker made his full debut in Return of the Jedi, he strode into Jabba the Hutt's palace and was stopped by two Gamori guards. Luke raised his hand and used the Force to disperse them, and the two guards backed away clutching their necks in pain. This means that Luke uses Darth Vader's signature dark side powers, and while the film's 1983 novel depicts a similar moment, its primary novel (which is both canon and part of the legend) and Age of Legends Heir to the Empire, Reveals Luke's Behavior Not actually connected to the dark side.
Star Wars Canon Reveals Luke Used A Jedi Mind Trick, Not Force Choke, In ROTJ

James Kahn's first novel, Return of the Jedi, is part of the legendary timeline and strongly hints that Luke used a Force choke on two guards, but, like the movie itself, doesn't make Luke's response to the dark The potential use of the power of the face becomes clear. The film, written by Ryder Windham's primary novel, was originally part of the saga but has since become part of the new canon, depicting Luke's powers as a kind of mind trick. World of Legends further confirms that Luke didn't use a forced choke, referencing Luke's entry into Jabba's palace in Heir to the Empire, where Luke simply confuses the guards.
Since both canon and legend confirm that Luke is not using a Force choke, it seems odd that Gamor guards would grip their necks in agony as if they were being choked by the Force, but there is a potential for this explanation of. Luke used a mind trick on the guards, but he probably just created the illusion of suffocating them rather than actually constricting their windpipes and potentially killing them. This preserves the look of Luke's dark side powers without betraying his role in the film.
Luke's Mind Trick Better Fits His Return Of The Jedi Arc

Although Luke struggles with the temptations of the dark side in Return of the Jedi, he only briefly touches on it in the third act after a prolonged period of seduction and a credible threat to his sister. Luke eventually resists the dark side before causing irreparable damage, but this moment is destined to be the only time he actually uses its power. Luke Skywalker's actions in the first act of Return of the Jedi are intended to be the actions of the hero saving his friends, so having him use the dark side, even a non-lethal force choke application, is the opposite of that intent.