You can open a private chat with a participant in a chat room with
M-x jabber-muc-private (or by using the MUC menu). This creates
a buffer with the name
*-jabber-muc-priv-group-nickname-* (customizable by
jabber-muc-private-buffer-format), which behaves mostly like an
ordinary chat buffer. This buffer will also be created if someone
sends a private message to you.
Private MUC messages use the same alerts as normal chat messages. See Message alerts.